A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraints

Constrained Markov decision processes (CMDPs) formalize sequential decision-making problems whose objective is to minimize a cost function while satisfying constraints on various functions. In this paper, we consider the setting of episodic fixed-horizon CMDPs. We propose an online algorithm which leverages linear programming formulation repeated optimistic planning for finite-horizon CMDP provide probably approximately correctness (PAC) guarantee number episodes needed ensure near optimal policy, i.e., with resulting value close that and within low tolerance, high probability. The shown have dependence sizes state action spaces quadratic time horizon upper bound possible successor states state-action pair. Therefore, if much smaller than size space, becomes in horizon.